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Introduction 


We live in a time of staggering 


technological change, with disruptive 
technology innovations continually 
reshaping how we live and work. One 


example is the indispensable role 
technology now plays in science, 
technology, engineering, and 


mathematics (STEM) professions. To 
prepare young people for citizenship 
and a workforce with rapidly changing 


requirements, we must design learning 
environments that provide all students with unique and engaging 
opportunities to master STEM skills using the technologies STEM 


professionals use. 


This is already occurring in our nation’s 
schools. Coding courses and workshops 
engage students in computational thinking. 3D 
modeling software and printers let students 
become “makers” solving real-world problems 
through design and engineering. Robotics 
competitions combine digital coding and 
design with physical building to help students 
learn collaboration and problem-solving skills. 
With augmented reality, students can now 
develop field research skills while exploring 
simulations of ecosystems in faraway places. 


Still, as with all educational innovations, 
bringing technology into classrooms alone 
does not ensure better outcomes. Powerful 
learning experiences also require effective 
teachers who make instructional decisions 
based on evidence of enhanced student 
learning. 


To help teachers implement new, research- 
based approaches for leveraging technology 
to improve STEM learning, the U.S. Department 
of Education, in collaboration with Digital 
Promise, conducted a systematic review of the 
research literature on the impact of integrating 


innovative digital technology in STEM and 
computer science curricula and classrooms. 
For purposes of the review, we defined digital 
technology broadly to include various forms of 
computing devices and software applications, 
including sensors connected to computing 
devices, digital immersive environments, 
cellular devices and mobile applications, as 
well as collaborative technologies. 


The research review identified nine dimensions 
of effective STEM learning practices enabled 
by technology, which are described in this 
report. The report also provides links to videos 
developed by the Department and Digital 
Promise that highlight the research-based 
dimensions in action in 10 spotlight schools 
(Exhibit 1). The purpose of the STEM spotlights 
is not to advocate for technology for its own 
sake, but rather to illustrate meaningful, 
research-based STEM learning experiences 
that can be supported with digital tools in 
classrooms across the country. 


An appendix provides additional detail on the 
methodology used for the research review. 
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Nine dimensions of powerful 
STEM learning 


The review of the literature surfaced nine dimensions of powerful 
STEM learning that can be supported with digital technology. The 
review looked at studies of supports for students’ development of 
STEM and computational thinking knowledge and practices (skills) 
that provided evidence of student learning outcomes, as measured 
by an objective test and not just self-reported perceptions of 
learning. The nine dimensions were checked for consistency with 
conceptual framings and empirical findings from the Department 
of Education's Policy and Program Studies Service and the Institute 
for Education Sciences, as well as key policy documents such as 
the National Science & Technology Council's 2018 report, Charting 
a Course for Success: America’s Strategy for STEM Education’. 

Our process also involved soliciting input from STEM education 
research experts. Below we present top-line summaries of the nine 
dimensions. 
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Engineering Design Processes. Students 
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The nine dimensions served as a framework to 
identify 10 schools that could be spotlighted 
for implementing one or more of the STEM 
dimensions in their classrooms. Spotlight 
schools were selected based on nominations 
from a Technical Working Group of leading 
education researchers and other individuals 
knowledgeable about technology use for 
powerful STEM learning. Also considered 

were schools recognized as models of STEM 
education such as Future of Education 
Technology's STEM Excellence Award finalists 
and winners, Intel Schools of Distinction Award 
finalists and winners, and schools profiled in 
policy reports such as Monitoring Progress 
Toward Successful K-12 STEM Education (NRC, 
2013). 


This process identified 100 schools, each of 
which were invited to submit an application 
that asked them to describe in detail a 
classroom lesson that used technology 

for addressing one or more of the STEM 
dimensions. Of these, 34 schools submitted 
applications and 15 were interviewed about 
their suitability as a spotlight school. During 
the selection process we sought to include 
a broad representation of geographic areas 
and a range of grade levels (i.e., elementary, 
middle, and high schools). Exhibit 1, below, 
shows the names and locations of the final 10 
spotlight schools. 
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Exhibit 1. Names and locations of the ten spotlight schools. 


Weaver Lake 


Henrietta Lacks Health and Elementary School Pine Grove 


Bioscience High School ; Maple Grove, MN Middle School 
Vancouver, WA Schoo Middle School East Syracuse, NY 
Lincoln, NE 
South Side 
Elementary School 
Kendallville, IN 
Design Tech High School 
Redwood City, CA \ 4d eee Greene Central 
High School 
(ay/ Snow Hill, NC 


= 
can 


Walter Bracken 
STEAM Academy 
Las Vegas, NV 


Douglas L. Jamerson, Jr. 
Elementary School 
St. Petersburg, FL 


Cedars Intl. Next 
Generation High School 
Austin, TX 


The following section provides more details about the nine dimensions and the 10 spotlight 
schools. Each dimension is supported with examples of powerful practice from the research 
literature and highlights an exemplary STEM lesson from one of the 10 spotlight schools.? 


2 Information for school “Quick Facts” obtained from the Institute of Education Sciences National Center for Education Statistics 
website. 
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1. Dynamic Representations. Students learn or 
master STEM concepts through interacting with digital 
models, simulations, and dynamic representations of 
mathematical, scientific, and engineering systems. 


Dynamic representations such as digital 
models, interactive simulations, and virtual 
environments are fundamental tools used by 
scientists, mathematicians, and engineers. 
Based on a computer model, dynamic 
representations engage learners and help 
them construct more accurate mental models 
of natural or engineered phenomena that 
would otherwise be difficult or impossible 

to observe.* Dynamic representations vary 
widely and differ in the levels of control and 
interactivity they provide to students. To better 
support learners, they are often augmented 
with pedagogical scaffolds that guide students 
when learning about STEM systems and 
processes. A study on middle school students’ 


physics learning, for example, found that 

there was a significant reduction in students’ 
misconceptions about motion and force after 
using a simulation-based learning environment 
that included such scaffolds.’ In high school 
mathematics, the interactive software SimCalc 
has been shown to significantly improve 

how students learn core algebra concepts 

that include procedural and conceptual 
problems.° In a meta-analytic study on 
simulations, researchers found that students 
using simulations for STEM learning performed 
better on measures of learning than students 
engaged in equivalent activities that did not 
involve simulations.°® 


Quick Facts 


Location: Lincoln, NE (rural, fringe) 


FRPL: 44% Asian: 3% 
Black 4% White: 73% 
Hispanic: 11% Other: 9% 


Watch the Walter Bracken STEAM Academy video! 


3 National Research Council (2011) 
4 Huang et al. (2017) 

5 Hegedus et al. (2015) 

° D'Angelo et al. (2014, March) 
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2. Collaborative Reasoning. Technology tools 
support students’ collaborative reasoning around 
STEM concepts, equalizing participation among group 
members and helping individuals and groups improve 


their ideas. 


Collaboration facilitates learning through 
students’ ongoing efforts to construct and 
maintain a shared understanding of a problem.’ 
When students work together to reason about 
STEM ideas, they engage in negotiation and 
renegotiation around meaning. Technology 
can enhance collaboration by expanding the 
kinds of exchanges students can have and by 
increasing access to community knowledge. 
Research has shown that students benefit 
more from collaborative activities when they 
receive support and guidance for interacting 
with one another.® In a study of fourth graders 
learning science, use of a digital collaboration 
platform that included e-sticky notes 


Next Generation Science Storylines 


facilitated richer collaboration and resulted in 
stronger responses to open-ended questions. 
Observations showed that the technology 
also appeared to help equalize participation— 
students were more encouraged to change 
their minds and were less likely to dominate 
the conversation or to acquiesce to other 
students dominating classroom interactions 
when using the collaboration platform.’ Fifth- 
graders who used Mindtool to create science 
concept maps that were shared and discussed 
with peers had higher achievement scores than 
students engaged in the same activity using 
hand-drawn maps.*° 


Quick Facts 


Location: Lincoln, NE (rural, fringe) 


FRPL: 44% Asian: 3% 
Black 4% White: 73% 
Hispanic: 11% Other: 9% 


Watch the Philip H. Schoo Middle School video! 


7Roschelle & Teasley (1995) 
8 Dillenbourg (1999) 

° Lan et al. (2010) 

10 Hwang et al. (2011) 
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3. Immediate, Individualized Feedback. Digital tools 
provide students practicing or learning STEM skills or 
concepts with immediate and individualized feedback, 
beyond right or wrong. 


Feedback is considered one of the most 
powerful means for influencing student 
learning." When accompanied by challenging 
goals, feedback informs learners about the 
level of performance that is required so they 
can set reasonable goals and direct their 
actions and effort accordingly.’ When learning 
goals are clear and students are committed, 
feedback can help students reduce the gap 
between their current level of performance 
and the intended level.** “ Feedback can take 
a number of different forms depending on the 
learning goals of a lesson. For example, when 
students are engaged in tasks such as test- 
taking, some delay in feedback is beneficial. 
When students are engaged at the process 
level, however, as when processing classroom 
activities, immediate feedback is more 


desirable. For example, a quasi-experimental 
study involving 4,000 second grade students 
from 129 classrooms found greater learning 
gains in basic calculation skills when students 
practiced their arithmetic skills using a 
computer and received immediate feedback 
about their response.'© Feedback on task- 
based activities can be provided at both the 
individual and the group level. When provided 
at the group level, learners must decide if the 
feedback is relevant to them individually, to 
the group as a whole, or to other individuals.”’” 
A study of fifth-graders found that students 
learned fractions faster when the software 
provided individualized feedback about 

their misconceptions beyond whether their 
response was correct.”® 


Quick Facts 


Location: Snow Hill, NC (rural, distant) 


FRPL: 96% Asian: 0% 
Black 43% White: 32% 
Hispanic: 25% Other: 1% 


Watch the Greene Central High School video! 


1 Hattie & Gan (2011) 

? Locke & Latham (1990) 

43 Hattie & Timperley (2007) 
4 Kluger & DeNisi (1996) 

5 Kulik & Kulik (1988) 

16 Beserra et al. (2014) 

7 Nadler (1979) 

18 Chu et al. (2014) 
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Science argumentation is a thought process 
that requires critical thinking to present and 
defend evidence that explains an idea about 
scientific phenomenon.? Argumentation is 
central to the practice of all fields of science. 
In the classroom, science argumentation 
enables students to construct knowledge 

and empowers them to criticize plausible- 
sounding ideas that lack supporting 
evidence.*°* Technology can promote science 
argumentation in a number of ways. A 2011 
study tested the impact of computer-based 
scaffolds for argumentation on middle school 
students’ ability to evaluate evidence-based 
arguments during problem-based learning. 
Components of arguments that were tested 
included making claims, providing evidence 
that supports the claim, and explaining the 


Weaver Lake STEM Elementary School 


AAt Weaver Lake STEM Elementary School in 


IME] eK i col em Nal aleve): Mm cele |adaMe| e-lel-lecmaler-lel— 
Tale WN'A=¥-1 ed Co) ale “ide (ehVar-] okeLU ial (-],<-m al-r-] damon (=t-] aq) 


Yel (=]alet=w- 1 ce [Ul aal=jaiecl ile) anc) (icmuriialemel-lt-i-lale| 
evidence. Guided by the question, “Is the lake 


4. Science Argumentation. Students use technology 
that supports science argumentation skills including 

Sy presenting and evaluating evidence about scientific or 
mathematical claims. 


relevancy of the evidence for supporting the 
argument in a claim. The researchers found 

a significant improvement among lower- 
achieving students’ ability to evaluate the 
quality of scientific arguments after using 

the computer-based scaffolds. A different 
study of 960 middle school science students 
found that students improved their science 
argumentation skills through an online, 
multiplayer game that was instructionally 
designed to integrate scientific argumentation. 
Students who played the game 10 times or 
more improved in all aspects of argumentation, 
including understanding a claim, judging 
evidence about a claim, determining the 
reasoning applied to the claim, and making 
judgments.” 


Quick Facts 
Location: Maple Grove, MN (suburb, large) 
FRPL: 35% Asian: 22% 
Black 26% White: 42% 


healthy?” students pay monthly visits to a nearby 


freshwater lake to collect environmental data 


Hispanic: 4% Other: 7% 


with digital tools used by real scientists. Students 

use probes to record water and soil temperatures; collect samples of lake water; and 

UKXer- In del ge}(oliavmdu] oX-mcol gree) ge Mr 1X1 arel (1a 9\Amcl de (el—alecw-|om t-], (mel (o]it-] Me) areiceye|e-]elame) meals 
lakefront to document seasonal observations at different times of the year. Back in the 
classroom, students are tasked to create a website that lets the broader school community 
Caen VA al=14 alt=) exe) am avelandal=mt-), (mim al=t-]idanvaey-ly-1e me) amdal=1/arel-]¢- mam Creal avelvelehva el-\aaaliemsieb (el lal es 
CoM) micron V=1NYe [i ol Nan val=m=aVilel=laler=m vale lai val-\vaece) |(=reid-Yoo)’{-1and al -mcxel alele)MY(-r-]am(larelorel late) 
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19 Berland & Reiser (2009) 
20 Berland & Reiser (2010) 
1 Ford (2008) 

22 Ault et al. (2015) 
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with different bird species. Students then get to create their own species of a bird by 
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the appropriateness of their bird for a selected habitat. Students use facts about birds as 
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technologies. 


Engineering can be defined as the process of 
designing for the human-made world.” Like 
science, engineering involves both iterative 
and systematic processes. Engineers test 

and modify each new version of a design 
based on available knowledge, and then 
systematically undertake steps to improve the 
design.19 Using data, engineers ask questions 
to understand the design problem, determine 
criteria for a successful solution, and identify 
constraints.” Digital tools play a central 

role in the engineering design process. In 
education, technology tools provide increased 
opportunities for students to apply scientific 
and mathematical ideas to their designs. Fan 
and Yu's (2017) quasi-experimental study of 
high school students found improvements 


23 National Academy of Engineering (2008) 
*4 Feder et al. (2009) 
*5 Guzey et al. (2017) 


5. Engineering Design Processes. Students plan, 
revise, implement, and test problem solutions using 
engineering design processes and appropriate support 


in STEM domain knowledge and higher 

order thinking skills, including problem 

solving abilities and procedural knowledge, 

for students who completed four units of 

an engineering module that integrated 
components of science and mathematics. 
Students constructed a cam mechanism toy 
using lever scales, a gear-wheeled range 
finder, and a gear sets and then used computer 
aided design (CAD) software to simulate and 
analyze motion and the speed ratio of different 
gears. In middle school, seventh-graders 
knowledge gains in engineering and math 
were higher when they used technology tools 
for data analysis and measurement within a 
curricular unit on heat transfer.*° 
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Quick Facts 
Location: St. Petersburg, FL (city, large) 
FRPL: 53% Asian: 3% 
Black 37% White: 48% 
Hispanic: 7% Other: 5% 


Watch the Douglas L. Jamerson, Jr. Elementary School video! 


Quick Facts”° 
Location: Redwood City, CA (suburb, large) 
FRPL: 11% Asian: 20% 
Black 0.4% White: 50% 
Hispanic: 13% Other: 17% 


Watch the Design Tech High School video! 


6 Information for Design Tech High School “Quick Facts” obtained from the California Department of Education website. 
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6. Computational Thinking. Students use technology 
to problem-solve using algorithms, data, and 
simulations to investigate questions and develop new 
understandings about phenomena. 


Computational thinking involves the conclusions from data.*° In the study, students 
formulation and analysis of problems and developed a computer program from source 
their solutions using methods and tools code based in algorithmic thinking and then 
for reasoning abstractly and automating used the program to solve a mathematics 
procedures through algorithmic thinking.*”° problem set. The code was transferred into 
In education, the practices learned through MATLAB to generate graphical representations 
computational thinking are widely applicable of the solution that illustrated the correctness 
to topics in science and mathematics and of the code and the solution of the exercise. 
provide a real-world and applied context A study of rural and indigenous middle school 
for working with abstract datasets.*° For students also showed learning gains from 
example, a quasi-experimental study with using robotics and game design that relied 
high school students found that teaching on abstraction skills, logical thinking, use of 
computational thinking in conjunction with algorithms, and analyzing and implementing 
computer programming resulted in greater solutions.*! Samples of student work were 
improvements in students’ reasoning skills, analyzed for evidence of computational 
including making conjectures and drawing thinking. 

Quick Facts 


Location: Kendallville, IN (town,distant) 
FRPL: 64% Asian: 0.2% 

Black 0.5% White: 84% 
Hispanic: 12% Other: 3% 


77 Basu et al. (2014) 

78 Grover & Pea (2013) 

22 Wing (2008) 

30 Psycharis & Kallia (2017) 

3 Leonard et al. (2016) Innovation Spotlights 14 


7. Project-based Interdisciplinary Learning. Students 
use digital technology tools in the context of authentic 
project- or challenge-based learning activities that 
integrate multiple STEM fields (e.g., science and 


mathematics). 


Research shows that using an interdisciplinary 
curriculum that integrates multiple STEM 
subjects provides students with a more 
relevant, more engaging, and less fragmented 
learning experience.” In the context of 
project- or challenge-based learning, students 
can pursue solutions to real problems in the 
same way that professional scientists do.* In 
this context digital technologies can be used to 
find, organize, and communicate information 
and ideas; to support task management and 
productivity; and to create a final product. 
When teachers use this instructional approach, 


Henrietta Lacks Health and Bioscience 


High School 


Henrietta Lacks (HeLa) Health and Bioscience 


High School in Vancouver, Washington provides 


students with interdisciplinary project- or 
challenge-based learning through a project in 
which students learn about the chemicals in 


sunscreen and their impact on the environment. 


students’ learning activities culminate in a 
tangible and sharable product that adequately 
addresses the problem or question.** The 

final product or artifact gives students an 
opportunity to publicly display their work and 
to practice presenting and explaining their 
project's outcomes to others. With project- 
based interdisciplinary activities, students 

not only learn important STEM content and 
practices, but also practice applying them in 
conjunction with other skills and knowledge in 
a realistic context. 


Quick Facts 


Location: Vancouver, WA (city, midsize) 


FRPL: 41% Asian: 9% 
Black 2% White: 66% 
Hispanic: 15% Other: 8% 


The challenge starts by creating a model of a molecule that’s commonly found in over- 
the-counter products including sunscreen. Through their research, students learn that 
certain compounds such as oxybenzone are harmful for coral reefs and for your skin. In 
their investigations students use the authentic and freely available PubChem database of 
chemical information and 3D interactive chemical structure models. For their culminating 
artifact, students create a project poster as well as a public information piece in the form of 
a sunscreen brochure. Students present their research to the entire class and describe how 
the active ingredients found in sunscreen affects coral reef systems and human health. 


Watch the Henrietta Lacks Health and Bioscience High School video! 


%2 Frykholm & Glasson (2005) 
33 Krajcik & Blumenfeld (2006) 
+4 Blumenfeld et al. (1991) 
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8. Embedded Assessments. Digital assessments are 
embedded in STEM instruction to prompt students’ 
reflection on the quality of their explanations, models, 


or problem solutions. 


When embedded in STEM instruction, digital 
assessments can provide rich information 
about the nature and quality of students’ STEM 
practices during scientific inquiry.* °° When 
used for formative purposes, assessments 
provide a valuable opportunity for adapting 
instruction to better meet learners’ needs.°*’ 
Such assessments are especially powerful 
when they fuse core STEM ideas with 

STEM practices. A study by Zaidi (2016), 

for example, investigated the impact of 
embedded formative assessments designed to 
promote integrated science knowledge. The 
assessment allowed students to receive both 


peer and expert feedback on their knowledge 
products, which students then used to revise 
and resubmit their products. Study findings 
showed that the formative assessment 
improved science learning for all students 
but was particularly effective for students 
from underrepresented groups. Moreover, a 
qualitative analysis of students’ knowledge 
artifacts revealed that students’ peer reviews 
became increasingly elaborate and specific 
over time, enabling them to produce higher 
quality integrated knowledge products. 


Quick Facts 


Location: East Syracuse, NY (rural, fringe) 


FRPL: 45% 
Black 4% 


Asian: 2% 
White: 87% 


Hispanic: 2% Other: 4% 


%5 Songer & Gotwals (2012) 
6 Songer et al. (2009) 
7 Black (2003, April) 
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9. Evidence-based Models. Students use technology to 
develop models based on data and evidence. 


Developing and testing evidence-based 
models is a core practice that scientists use 
when investigating theories about the natural 
world. Engineers likewise rely on evidence- 
based models when they design systems and 
solutions that solve practical problems.*° To 
develop a deep understanding of the practices 
that STEM professional use, students need 

to develop their own models based on data 
and evidence and the capacity to explain 

the evidence base and limitations of those 
models.*’ A clear example of how middle 
school students might use technology for 
developing and revising models is described 
by Krajcik and Merritt (2012). Sixth-grade 
students, for example, can use tablets or 


another technology tool to draw a model of 
how odor travels to the nose when learning 
about the movement of gasses and liquids. 
Students revise their model multiple times 
during an eight-week unit, making adjustments 
to the representation as they gain knowledge 
through inquiry activities where they explore 
the properties of gasses. As the unit progresses 
and students’ knowledge increases, their 
models gradually become more consistent 
with the full scientific model. STEM instruction 
that fuses science content and practices in this 
way supports students in developing necessary 
skills to understand and engage in scientific 
inquiry process. 


Quick Facts 


Location: Austin, TX (city, large) 


FRPL: 48% Asian: 5% 
Black 20% White: 15% 
Hispanic: 57% Other: 2% 


Watch the Cedars International Next Generation High School video! 


38 National Academy of Engineering and National Research Council (2014) 


3 Krajcik & Merritt (2012) 
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Conclusion 


Technology plays a central role in STEM professions and our 
everyday lives. So, while a strong STEM education supported by 
technology is valuable for young people who intend STEM careers, it 
is essential that everyone has access to STEM learning opportunities 
enhanced by technology that prepare them for life and work in 
today’s complex world. 


Numerous opportunities for STEM learning with technology exist 

in both formal and informal educational experiences at all levels of 
our education system. However, as in all fields of study, technology 
alone is not sufficient to make a difference in STEM. For powerful 
learning to occur, teachers must be able to make instructional 
decisions based on evidence that a particular approach will improve 
outcomes. 


A systematic review of the research literature on the impact of 
technology in STEM learning, summarized in this report, identified 
nine dimensions practices that result in powerful learning. The nine 
dimensions, with examples of ways in which they are being used in 
classrooms across the country, can help teachers make effective 
decisions about how to support STEM learning with technology now 
and in the future. 
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Appendix: Methodology 


When reviewing the scholarly literature, we focused our efforts on research that uses an 
experimental or quasi-experimental design to test the impacts of an intervention (e.g., a curricular 
unit or instructional approach) on student learning outcomes. We also included studies that 

were suggested by recognized experts in the field of STEM education.*° A number of the Pathway 
Objects described in the Charting a Course for Success report further supported the STEM 
dimensions. These include “Make computational thinking an integral element of all education” 
(Dimension 6), “Encourage transdisciplinary learning” (Dimension 7), and “Expand digital platforms 
for teaching and learning” (Dimensions 1-9). 


Identification of relevant studies 


In our search we consulted the EBSCO, ERIC, and ProQuest databases along with the search 
engine Google Scholar. The Stanford University librarian recommended these databases because 
they are the largest resource collections relevant to education. Scholarly peer-reviewed work 
and grey literature (e.g., conference proceedings, dissertations and theses, technical reports, 
white papers, etc.) that were published in English between 2010-2018 and listed “education” as 
the subject (keyword) were included in the search. Additional search criteria included one or 
more keywords from each of the categories listed in Table 1, below. We also examined resources 
produced by the Department of Education's Policy and Program Studies Service (PPSS), the 
Institute for Education Sciences (IES) and sources cited by key STEM policy documents. For PPSS, 
every resource within the “Educational Technology” subsection within the “K-12 Instructional and 
Improvement Strategies” section was reviewed for relevance. For IES, every resource from 2010- 
2018 that contained “technology” in the title was reviewed for relevance. 


40 We are grateful to the individuals who supported this work by reviewing and providing guidance on the nine STEM dimensions 
and/or provided suggestions for potential spotlight schools: Nancy Butler-Songer (Drexel University), Jennifer Corn (NC State 
University), Bill Penuel (University of Colorado, Boulder) and Joseph Krajcik (Michigan State University). 
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Table 1. Search keyword list 


School level Subject area* Digital technology 
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i (Yaalyale-layg Mathematics Digital games 
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Se a Technology 2D or 3D design software 
3D printer 
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Virtual labs/reality/field trips 
Are (=YommVi(el=rem-velidiale) 
sole (ot- ki war [U(e |e g=xexo)ce|lale) 
Makerspace 
Mobile devices 
Noy et (or-} ure) aly] 0) ods 
Al or Augmented reality 


Tate} fatex-alaremccvolalaveltoye Nap 
fol=SJle Tal 


*Among studies identified with these keywords, we will prioritize those that include the phrase 
“project-based learning,” “problem-based learning" and/or “inquiry-based learning.” 


Coding of eligible studies 
Studies that were coded for the research review were first screened according to established 


criteria. To be eligible for inclusion a study needed to: 


e Examine students’ learning or mastery of STEM content or practices through using/ 
integrating one or more digital technologies in K-12 STEM and/or computer science 
classroom contexts; 


e Report learning outcomes in STEM and/or computer science using an objective measure 
(see Table 2); and 


e Provide enough statistical information to support a judgment of whether or not the study 
would meet WWC standards with or without reservations.** 


4. Studies that would not meet WWC standards were not discarded, but were flagged. Each study was coded as “Appears to meet 
WWC standards,” “Appears to meet WWC standards with reservations,” or “Does not meet WWC standards.” 


Innovation Spotlights 28 


Studies that were retained for the review after screening were coded using a study review 
template that was designed for this purpose. When selecting studies to include in the review, 
attention was paid to the following: 


e Inclusion of studies that represent a broad range of student subgroups including rural, 
urban, minorities, girls, and historically underrepresented groups in STEM, etc.; 


e Inclusion of studies with positive/successful outcomes when reviewing studies with an 
objective assessment; and 


e Studies where technology was used to directly support students’ development of STEM/ 
computer science knowledge or practices (see Table 2 for general categories). 


Table 2. Descriptions of STEM/computer science learning outcomes 


STEM dimension Sample outcomes 
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Results from study search 


Our database and Google Scholar search yielded 264 articles (see Figure 1, below). The PPSS, IES, 
and STEM policy document review added four references. A review of the studies based on our 
eligibility criteria narrowed the research corpus to 64 studies that tested the effectiveness of a 
digital intervention in supporting an objectively measures STEM learning outcome. These studies 
were reviewed and discussed by the team to derive a tentative set of dimensions for the rubric. 


Figure 1. Study search process for research synthesis 


Studies identified Studies identified 
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abstiact serecning e Full-text paper not available 
aie Neer 
alxavc=Jaidcelamatelmelfe live] 
(@foy aa] ek-lavolamero) ale lidiolam ave) ae (=sxe1g el=\0| 
Studies remaining after Did not measure learning 
full-text screening . Learning outcomes were student- or 
(n = 66) teacher- reported 


NS} dU [ol (=m eavore) aalaal=1are(=te ll ony, 
experts (2). Total studies 
Tavcelaaaliare esa Um B)ieat=lalilelals 
(amet) 


In our review we sought to identify studies that clearly described the link between technology 
use and students’ STEM learning outcomes. Ideally, a study would provide enough description of 
the treatment condition to explain the role that technology played in facilitating students’ STEM 
learning. These descriptions informed a number of emergent themes that ultimately became the 
first seven practices for supporting powerful STEM learning with technology. Feedback received 
from experts resulted in the addition of dimensions 8 and 9. Together with the spotlight videos, 
the nine dimensions provide teachers and school leaders with tangible examples of effective 
technology use for supporting powerful STEM learning. 
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